Search for: All records

Preferences within a group of people are not uniform but follow a distribution. While existing alignment methods like Direct Preference Optimization (DPO) attempt to steer models to reflect human preferences, they struggle to capture the distributional pluralistic preferences within a group. These methods often skew toward dominant preferences, overlooking the diversity of opinions, especially when conflicting preferences arise. To address this issue, we propose Group Distributional Preference Optimization (GDPO), a novel framework that aligns language models with the distribution of preferences within a group by incorporating the concept of beliefs that shape individual preferences. GDPO calibrates a language model using statistical estimation of the group's belief distribution and aligns the model with belief-conditioned preferences, offering a more inclusive alignment framework than traditional methods. In experiments using both synthetic controllable opinion generation and real-world movie review datasets, we show that DPO fails to align with the targeted belief distributions, while GDPO consistently reduces this alignment gap during training. Additionally, our evaluation metrics demonstrate that GDPO outperforms existing approaches in aligning with group distributional preferences, marking a significant advance in pluralistic alignment. 

Finetuned large language models (LLMs) have shown remarkable performance in financial tasks, such as sentiment analysis and information retrieval. Due to privacy concerns, finetuning and deploying financial LLMs (FinLLMs) locally are crucial for institutions and individuals. In this paper, we employ quantized low-rank adaptation (QLoRA) to finetune FinLLMs, which leverage low-rank structure and quantization technique to significantly reduce computational requirements while maintaining model performance. We also employ data and pipeline parallelism to enable local finetuning on commodity GPUs. Experiments on financial datasets validate the efficacy of our approach in yielding notable improvements over the base models. 

Abstract If Type Ia supernovae (SNe Ia) result from a white dwarf being ignited by Roche-lobe overflow from a nondegenerate companion, then as the SN explosion runs into the companion star its ejecta will be shocked, causing an early blue excess in the lightcurve. A handful of these excesses have been found in single-object studies, but inferences about the population of SNe Ia as a whole have been limited because of the rarity of multiwavelength follow-up within days of explosion. Here we present a 3 yr investigation yielding a nearly unbiased sample of nine nearby (z < 0.01) SNe Ia with exemplary early data. The data are multiwavelength, coveringUBVgriand Neil Gehrels Swift Observatory UV bandpasses, and also early, with an average first epoch 16.0 days before maximum light. Of the nine objects, three show early blue excesses. We do not find enough statistical evidence to reject the null hypothesis that SNe Ia predominantly arise from Roche-lobe-overflowing single-degenerate systems (p= 0.94). When looking at the objects’ colors, we find the objects are almost uniformly near-UV–blue, in contrast to earlier literature samples which found that only a third of SNe Ia are near-UV–blue, and we find a seemingly continuous range ofB − Vcolors in the days after explosion, again in contrast with earlier claims in the literature. This study highlights the importance of early, multiwavelength, high-cadence data in determining the progenitor systems of SNe Ia and in revealing their diverse early behavior. 

Large Vision-Language Models (LVLMs) have made substantial progress by integrating pre-trained large language models (LLMs) and vision models through instruction tuning. Despite these advancements, LVLMs often exhibit the hallucination phenomenon, where generated text responses appear linguistically plausible but contradict the input image, indicating a misalignment between image and text pairs. This misalignment arises because the model tends to prioritize textual information over visual input, even when both the language model and visual representations are of high quality. Existing methods leverage additional models or human annotations to curate preference data and enhance modality alignment through preference optimization. These approaches are resource-intensive and may not effectively reflect the target LVLM's preferences, making the curated preferences easily distinguishable. Our work addresses these challenges by proposing the Calibrated Self-Rewarding (CSR) approach, which enables the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. In the reward modeling, we employ a step-wise strategy and incorporate visual constraints into the self-rewarding process to place greater emphasis on visual input. Empirical results demonstrate that CSR significantly enhances performance and reduces hallucinations across twelve benchmarks and tasks, achieving substantial improvements over existing methods by 7.62%. Our empirical results are further supported by rigorous theoretical analysis, under mild assumptions, verifying the effectiveness of introducing visual constraints into the self-rewarding paradigm. Additionally, CSR shows compatibility with different vision-language models and the ability to incrementally improve performance through iterative fine-tuning.